Tail Latency in ZooKeeper and a Simple Reimplementation

ZooKeeper [1] is a commonly used service for coordinating distributed applications. ZooKeeper uses leader-based atomic broadcast for writes, so that all state modifications are globally totally ordered, but it allows stale reads from any server for high read availability. This design trades high read throughput for potentially high write latency. Unfortunately, the extent of this tradeoff and the magnitude of ZooKeeper’s tail latency is unclear. Although [1] presents average request latency, tail latency is often far more important in real world applications [2] [3]. In this paper, we remedy this lack of information with three experiments. As a baseline, we also implemented a system called Safari [9] with the same API as ZooKeeper, but specifically designed for lower worst-case write latency. We examine the two systems’ latency characteristics in a single-machine and two realistic production environments. We also offer explanations for their performance differences and design tradeoffs, as well as some comments on practically deploying ZooKeeper and Safari.